When the loss is set as squared loss, we can modify the functions regarding the truncated spectral projection.

Modern distributed training of machine learning models often suffers from high communication overhead for synchronizing stochastic gradients and model parameters.

In this natural setting, it is not common to see the same data sample multiple times, or even twice at times.

Our proposed formulation for neuron calibration is lightweight and applicable to general feed-forward neural networks-based models.

What sets this setting apart from traditional stochastic optimization is the difference between merely updating model parameters and deploying the new model.

Overparameterized models with millions of parameters have been hugely successful. In this work, we ask: can the need for large models be, at least in part, due to the \emph{computational} limitations of the learner? Additionally, we ask, is this situation exacerbated for \emph{robust} learning? We show that this indeed could be the case. We show learning tasks for which computationally bounded learners need \emph{significantly more} model parameters than what information-theoretic learners need. Furthermore, we show that even more model parameters could be necessary for robust learning.